FUNK, COSINE, AND SINE TRANSFORMS ON 
STIEFEL AND GRASSMANN MANIFOLDS 



B. RUBIN 

Abstract. The Funk, cosine, and sine transforms on the unit 
sphere are indispensable tools in integral geometry. They are also 
known to be interesting objects in harmonic analysis. The aim 
of the paper is to extend basic facts about these transforms to 
the more general context for Stiefel or Grassmann manifolds. The 
main topics are composition formulas, the Fourier functional re- 
lations for the corresponding homogeneous distributions, analytic 
continuation, and explicit inversion formulas. 
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1. Introduction 

1.1. History and motivation. Our consideration has several sources. 

1. For a function $ on the unit sphere S 2 in R 3 P. Funk [Fill 
IF13] defined a circle-integral function (die Kreisintegral-Funktion) \ 
on the set of great circles as an integral of $ over the corresponding 
great circle. He suggested two inversion algorithms; see [F131 pp. 285- 
288]. The first one relies on expansion in spherical harmonics and the 
second reduces the problem to Abel's integral equation. From these 
results Funk derived the celebrated Minkowski's theorem [Mm], |HellO[ 
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p. 137], stating that bodies of constant circumference are bodies of 
constant width. 

2. Funk's Kreisintegral-Funktion is now called the Funk transform. 
This concept extends to higher dimensions, when great circles on S 2 are 
substituted by cross-sections S™^ 1 fl £, £ being a k- dimensional linear 
subspace of IR n , 1 < k < n — 1. The set of all such subspaces forms a 
Grassmann manifold G n j~ = 0(n)/(0(k) x 0(n — k)). These transfor- 
mations were extensively studied in Gelfand's school, by S. Helgason, 
and other authors; see [GGG1 lHel90l IHelOOl IHelTOl lRu98l lRu02l lRu03j 
and references therein. Further generalization connects functions on 
two different Grassmannians by inclusion, namely, 

(1.1) {R k ,if)(v) = [ f(0 d n £, £ e G n , k , r, e G n , e , k < £, 

where d^ denotes the probability measure on the Grassmann mani- 
fold Gk{ff) of all A;- dimensional linear subspaces of rj. Operators (11.11) 
are also known as Radon transforms for a pair of Grassmann mani- 
folds. T]iey were studied by Petrov [P67] . Gelfand and his collabora- 



tors [GGS701IGGR] . Grinberg [GH], Kakehi |K£], Grinberg and Rubin 
[GRj . Zhang \Zhl\ [Zh2j. Affine versions of (II. ip were considered in 
pOlO^lRuOillShi] . 

In spite of the elegance and ingenuity of the inversion methods in 
these publications, the resulting formulas are pretty involved. It is a 
challenging problem to find new simple inversion formulas and develop 
a theory which is parallel to that for the unit sphere. The present paper 
is devoted to this problem. The main idea of our approach is to treat 
operators (II. ip as members of the analytic family of the higher-rank 
cosine transforms, which will be introduced below. 

3. The name cosine transform was given by Erwin Lutwak [Lul p. 
385] in 1990 to the integral operator 

(1.2) (C/)(u)= / f(v)\u-v\dv, ueS n -\ 

Since then, this term is widely used in integral and convex geometry (in 
parallel with its traditional meaning in the Fourier analysis). Operator 
(11.21) and its generalization with the kernel \u-v | A were studied without 
naming by many authors in geometry and analysis since Blaschke [Bla], 
Levy [Lev] . Aleksandrov jSIj; see [Ga| IK97| IRu03] for references. A 
remarkable fact, that amounts to the results of Gelfand- Shapiro [GSha] 
and Semyanistyi [Se63j, is that cosine transforms are restrictions to 
gn-i Q £ Fourier transforms of homogeneous distributions on W 1 . 
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Specifically, if we set 

(£ a f)(u)= [ fiv^u-vr'dv, {74>){y) = / <t>{x)^ y dx 



n-l 



/ E L 1 ^- 1 ), </> E S(R n ), then 

(£ a f)(y/\y\) 

r (a/ 2 ) 7 R n \y 



i-3) ttttox / v r.:.!-a — 



C = 2 n+Q 7T rt - 1/2 . 



r((i-a)/2)y R „ ixi^- 1 

Integrals in this equality converge absolutely when < Re a < 1 and 
extend by analyticity to all a G C. An important observation due 
to Semyanistyi is that the Funk transform and its inverse are mem- 
bers of the analytic family of suitably normalized cosine transforms 
£ a . This result was extended in [Ru02j to the Funk- Radon transforms 
over totally geodesic submanifolds of arbitrary dimension. The corre- 
sponding cosine transforms have found application in convex geome- 
try |Ru08t IRulOat IRZ] . Some ideas from |Se63] were rediscovered by 
Koldobsky, who found remarkable application of the relevant Fourier 
transform technique to geometric problems; see [K05J and references 
therein. 

4. In the last two decades a considerable attention was attracted to 
generalization of the cosine transform for functions on the Stiefel and 
Grassmann manifolds. The impetus was given in stochastic geometry 
by Matheron |Mat[ p. 189], who conjectured that the higher-rank ana- 
logue of (jl.2p (the precise definition is given later) is injective. Math- 
eron's conjecture was disproved by Goodey and Howard [GH1], using 
topological results of Gluck and Warner |GWj . A self-contained Fourier 
analytic proof, versus [GHlj . was suggested by Ournycheva and Rubin 
|OR05a[ IOR06] . Goodey and Zhang |GZ] applied higher-rank cosine 
transforms to the study of lower dimensional projections of convex 
bodies; see also |Goot |Spo|. Interesting connections to group repre- 
sentations can be found in the papers by Alesker and Bernstein [ABJ, 
Alesker [A"] , and Zhang |Zh2j . One should also mention fundamental 
publications by Blind, Herz, Faraut, Khekalo, Rais, Petrov, Ricci and 
Stein, Stein, and others. They are devoted to analysis of homogeneous 
distributions on matrix spaces; see |Blilt IBli2t IFK} IHerz} IKhlat \Kh2\ 
IUR05al IUR061 lP70l iRaH [RSI ISt2] and references therein. 

1.2. Plan of the paper and main results. Below we give a brief 
account of main results and ideas of the paper. Further details and 
more results can be found in respective sections. Section 2 contains 
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preliminaries. We recall basic facts about matrix spaces, Radon trans- 
forms over matrix planes, Riesz distributions, and the composite power 
function associated to the cone Q of positive definite symmetric mxm 
matrices. 

In Section 3 we define the higher-rank Funk transform F m ^ as an 
operator (13. 2p that sends functions on the Stiefel manifold V n ,m of or- 
thonormal m-frames in IR n to functions on V nt f-, where m and k may 

be different. We also define the dual transform F m ,k and establish 
connection between our transforms and Radon transforms (11.11) . 

Theorem 1.1. Let l<m<k<n — m. The mapping f — > F m ^f has 
a kernel 

kerF m , fc = {/ G L l (V n ^ m ) : / f(vj) = a.e.}. 

JO(m) 

This theorem generalizes the well-known result in |HellO] , where the 
case m = 1 is considered for C°° functions. 

In Section 4 we introduce analytic families of non-normalized cosine 
and sine transforms for a pair of Stiefel manifolds; see (I4.1l) - (l4.4p . Nec- 
essary and sufficient conditions are obtained for these transforms to 
be represented by absolutely convergent integrals. Useful composition 
formulas, which generalize to m > 1 the corresponding equalities for 
m = 1 in |Ru02l IRu08] , are derived. One of them is 

(1-4) F m , k F m , k f = ~cQ n ~ h - m f, 

where Q a f, a = n — k — m, is the sine transform of / and a constant c is 
explicitly evaluated. This formula is well-known for Radon transforms 
of different kinds, where the operator on the right-hand side stands 
for the relevant version of the Riesz potential; see, e.g., [Radl |Fug 



IHellO} IRu06] . In the higher-rank case an analogue of the composition 

* 

F m ,kF m ,k played a crucial role in [GriJ, however, it was not explicitly 
computed and presented only in the spectral form on highest weight 
vectors. 

Section 5 is devoted to the Fourier analysis and analytic continuation 
of the cosine transform. Given A = (Ai, . . . , A m ) G C m , we introduce 
the composite cosine transform 

ipyu ) yv wc'"' ' X 

'V njk 

where (-) A denotes the power function of the cone Q, ""' stands for the 
transposed matrix, and integration is performed against the invariant 
probability measure on V n ^. Injectivity of such operators in the case 



1.5) (T* m ip)(v)= v(u)(v'uu'v) x d*u, v G V n . 
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k = m was studied in |OR05at IOR06] . For ( 11. 5ft and for the dual 

Funk transform F m ,k we obtain an analogue of the Fourier functional 
equation (II. 3p ; see Theorems 15 .41 and 15 . 1 01 An important new feature is 
that a function / on the right-hand side must be replaced by a certain 
complementary Radon transform of /. The latter boils down to the 
identity operator, when k = m. The Fourier functional equation for 
the operator ( 11.51) provides complete information about meromorphic 
structure of distributions of the form 

(1.6) a^(T«J,u), wen^), fEL\V n , m ), 

where k stands for the cosine or sine transform under considera- 
tion. We conjecture that if / G C°°(y n>m ) then the polar set of the 
distribution (II. 6p coincides with that of its pointwise counterpart 

a -)> (T™ k f)(u) for each u G V m ,k- 

To the best of our knowledge, a proof of this fact represents an open 
problem. 

In Section 6 we define normalized versions of the cosine and sine 
transforms. One of the main results of this section is Theorem 16. 6| 
according to which an integrable right 0(m)-invariant function / on 
V n>mi can be reconstructed in the sense of distributions from its Funk 
transform ip = F m ^f by the formula 

(1.7) a.c (i% l)k <p,u) = x k {f,u), ueC°°{V n , m ), 
where 

&mk ( P)(v) = 6 rhmtk (a) [ cp(u) (det(v'uu'v)Y a - k ^ 2 d*u, v G V n , m , 

Jv n>k 

is the normalized dual cosine transform and the constant x k is explic- 
itly evaluated. Thus, an inverse Funk transform is actually a member 
of the analytic family of suitably normalized dual cosine transforms, 
and all possible inversion formulas for the Radon transform ( II .ip on 
Grassmannians can be regarded as different realizations of the analytic 
continuation in ( ETJj H Since (f,co) = Vw G C°°(V ntm ) implies / = 
a.e., the validity of Theorem 11.11 follows; see also Remark 13. II 

Similar inversion results are obtained for the cosine transforms. An 
interesting observation is that, if k = m, then the Funk transform, 
the cosine transforms, and their inverses are (up to normalization) 
members of the same analytic family of operators. This fact was known 

1 This statement does not work for the restricted Radon transform as in |GGS70l 
IGGRj . because the latter is, in fact, another operator. 
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before only for m = 1 and established using decomposition in spherical 
harmonics. 

In Section 7 we suggest new realization of the inverse sine and Funk 
transforms in terms of powers of the Cayley-Laplace operator A = 
det(d/dx it j) in the space 9Jl niTn ofnxm real matrices. These powers 
are associated with the Riesz potential. The resulting formulas differ 
in principle from those in |GGR| IGri} \GR\ \K&\ IP67] . They look much 
simpler, however, must be interpreted in the sense of distributions. For 
instance, if we write x G Tt n!m in polar coordinates x = vr 1 ^ 2 , r £ £1, 
v ^ V n>m , and denote (E\f)(x) = (det(r)) A//2 /(t>), then, in the case of 
n — k — m even, we have 

f(v) = Cm,k (A -E_fc Fm,k<^)(v), ip = F mtk f, i = (n - k - m)/2, 

where the constant c m ^ is explicitly evaluated. If n — k — m is an odd 
number, then fractional powers of A are implemented, and the resulting 
inversion formula is non-local; see Theorems 17.21 and 17.71 In the case 
m — 1 inversion formulas of this kind were suggested by Semyanistyi 
[Sc63j and used in [Str81j IRu02j . The reasoning from those papers is 
unapplicable when m > 1. To get around this difficulty, we apply the 
relevant tools of harmonic analysis and several complex variables. 

We conclude the paper by Appendix containing evaluation of an aux- 
iliary integral and some comments on the celebrated paper |GGR] by 
Gelfand, Graev, and Rosu. The latter is devoted to the Radon trans- 
form (11.11) for a pair of Grassmannians. A keen reader can recognize 
our higher-rank cosine transform in IGGK pp. 367, 368]. However, 
some important details in that paper are skipped. We reproduce the 
relevant calculations and explain basic difficulties. 

Acknowledgements. I am indebted to many people for useful and 
pleasant discussions related to this paper. My special thanks go to 
Professors Tomoyuki Kakehi, Gestur Olafsson, Elena Ournycheva, An- 
gela Pasquale, Fulvio Ricci, Genkai Zhang. I am grateful to Susanna 
Dann who carefully read the original Funk papers. Her help was very 
valuable. 

2. Preliminaries 

In this section we establish our notation and recall some basic facts; 
see |UR04l IUR05al IOR06j for more details. 

2.1. Notation and conventions. Given a square matrix a, \a\ stands 
for the absolute value of det(a). We use standard notation 0(n) and 
SO(n) for the orthogonal group and the special orthogonal group of 
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R n , respectively, with the normalized invariant measure of total mass 
1. The abbreviation "a.c." denotes analytic continuation. 

Let 9JT njm ~ R nm be the space of real matrices x = (xij) having n 
rows and m columns; dx = Yl^iTYjLidXij', x' is the transpose of x, 
\x\ m = det(x'x) 1 / 2 , I m is the identity m x m matrix, and stands for 
zero entries. 

The Fourier transform of a function ip G L 1 (SD r l njm ) is defined by 

(2.1) 0{y) = {JF<p){y) = [ e'^^x) dx, y G M n , m . 
The relevant Parseval equality has the form 

(2.2) (0, 0) = (2n) nm (cp, 0), {<p, 0) = / <p{x)J(x)dx. 

This equality with in the Schwartz class i>(SEJt njTn ) of rapidly decreas- 
ing smooth functions is used to define the Fourier transform of the 
corresponding distributions. If G S{yfl n>rn ) and = J-" _1 is the 
inverse Fourier transform of 0, then, clearly, 

(2.3) 0(x) = (27r)- nm 0(-x), [0] A (x) = (2vr) nm 0(-x). 

Let S m ~ R m ( m + 1 )/ 2 be the space ofmxm real symmetric matrices 
s = (sij); ds = Yli<j dsij. We denote by Q = V m the cone of positive 
definite matrices in S m ; Cl is the closure of Q. For r G Q (r G Q), 
we write r > (r > 0). Given a and b in S m , the inequality a > b 
means a — b G and the symbol f(s)ds denotes the integral over 
the set (a+$7) n (b — Q). The group G = GL(m, R) of real non-singular 
m x m matrices g acts transitively on Q by the rule r — )■ grg'. The 
corresponding G-invariant measure on Q is [Tj p. 18] 

(2.4) d*r = \r\- d dr, d=(m + l)/2. 

If T m is the group of upper triangular mxm real matrices t = (Uj) with 
positive diagonal elements, then each r G Q has a unique representation 
r = ft. 

The Siegel gamma function of Q is defined by 

„ 171 — 1 

(2.5) r m (a)= / exp(-tr(r))|rf4r = vr^" 1 " 1 )/ 4 J] r(a-j/2). 

3=0 

This integral is absolutely convergent if and only if Re a > d — 1 = 
(m — l)/2, and extends meromorphically with the polar set 

(2.6) {(m-l-j)/2: J = 0,l,2,...}; 
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see [Gl], |FK] . [T] . For the corresponding beta integral we have 
(2.7) / \r - a\ a ~ d \b - r\ p ~ d dr = B m (a, (3) \b - a\ a+ ^ d 



fl;(a,/?) = r ; ( ° )r ;^ , itea>d-l, Rej3>d-1. 

For n > m, let V^ m = {f G 9^n,m : u'v = J m } be the Stiefel manifold 
of ortho normal m- frames in R n . This is a homogeneous space with 
respect to the action V n ,m, 3 v — >■ 7U , 7 G O(n), so that V^ i?n = 
0(n)/0(n — m). We fix a measure on V njm , which is left 0(n)- 
invariant, right 0(m)-invariant, and normalized by 

(2.8) cr n = / dv 



v n , m r m (n/2)' 

|Muj p. 70]. The notation d*v = a'^dv is used for the corresponding 
probability measure. 

We denote by G n ^ m the Grassmann manifold of m-dimensional lin- 
ear subspaces £ of R™ equipped with the 0(n)-invariant probability 
measure g?*£. Every right 0(m)-invariant function f(v) on \4 m can 
be identified with a function /(£) by the formula /({f}) = /(v), 
if} = t>R m G G„ m , so that L = f T/ f(v)d*v. Another 

identification is also possible, namely, / ({t> } ± ) = /(f), {f} 1 " G G n . n - m - 
We will be dealing with several coordinate systems on 2Jt n/m and V n ^ m . 

Lemma 2.1. (The polar decomposition). Let x G 9Rn,m, n > m. If 
rank(x) = m, then 

(2.9) x = vr 1/2 , v G V n>m , r = x'x G P m , 

and dx = 2~ m \r\^- m -^/ 2 drdv . 

For this statement see, e.g., [Herzj . |Muj . |FKj . Decomposition (12. 9 \ 
(but with r G 0) is valid for any matrix x G 9Jt n ,m> cf. |Muj p. 589]. 

Lemma 2.2. ( |P70| . |Ru06j ) If x E 9Jl n ,m> rank(x) = m, n >m, then 

x = vt, v E V n ,mi t E T m , 

and 

m 

dx = JT^J' ? dt*dv, dt* = Y\ dtij. 

j=l i<j 



Lemma 2.3. (The bi-Stiefel decomposition). Let k, m, and n be 

positive integers satisfying 

1 < k < n — 1, 1 < m < n — 1, k + m < n. 
(i) Almost all matrices v G V^ jm can 6e represented in the form 



(2.10) 

so t/jat 
(2.11) 



w(J m — a'a) 1 / 2 



a g 9Kfc, m , m g K-fc,-, 



f{v)dv 



dfi(a) 



f 



u(I m — a' a) 



'n.W 2 



dfi(a) 



0<a'a<I„ 



\I m — a'a\ da, 8 = (n — k)/2 — d, d = (m + l)/2. 
(ii) If, moreover, k > m, then 

wr 1 / 2 



(2.12) J f{v)dv = J dv{r) J dw J f 



dw, 



aV(r) = 2- m |r| 7 |/ m - r|*dr, 7 = Jfe/2 - d. 

For k = m, this statement is due to |Herzt p. 495]. The proof of 
Herz was extended in [GRJ to all k + m < n and simplified in |Zhl] ; 
see also |01R] . 

Lemma 2.4. Lei u G V^fc, t> G V n ^ m ; 1 < k,m < n. If f is a function 
of m x k matrices, then 



(2.13) 



f{v'u)d*u= I f{v'u)d*v. 



n,k 



Vn, 



Proof. We should observe that formally the left-hand side is a function 
of v, while the right-hand side is a function of u. In fact, both are 
constant. To prove f)2.13p . let G = 0(n), g G G, g\ = g^ 1 . The 
left-hand side is 



G 



f(v'gu)dg= / f ((g^)' u) dg r , 

JG 

which equals the right-hand side. □ 
The following statement is a particular case of Lemma 2.5 from |GRj . 
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Lemma 2.5. Let A G Wl k ,e, S = A' A G Vi, £ <m < k, m + £ <k, 
5 = (£ + l)/2, c = 2^a k _ m/ a m/ /a k/ . 

Then 

(2.14) [ f(A'u) 

= c\S\ 5 ~ k ' 2 f ' \S-sf- m ^ 2 - 5 \s\ m l 2 - 5 ds [ f{s l l 2 0)dj. 

Jo JV mt i 

2.2. The composite power function. Let Vt be the cone of positive 
definite symmetric mxm matrices. Given r = (r i; j) G Q, let A (r) = 1, 
Ai(r) = rx i, A 2 (r), . . ., A m (r) = |r| be the corresponding principal 
minors, which are strictly positive. For A = (Ai,...,A m ) G C m , the 
composite power function of the cone Q is defined by 

(2.15) n x{,) 



i=i 



Ai_i(r) 



A 1~ A 2 



A m— 1 A m 



= A^r)-^-...^.!^) 2 A m (r) : 
If r = ft, i = (tij) G T m , then 



(2.16) 



j=i 



This implies the following equalities: 

(2.17) r A+/x = r A r", r A+ao = r x \r\ a/2 , a = (a, . . . ,a); 

(2.18) (t'rt) A = (t't) x r x , tGT m . 

The reverses of A = (A l5 . . . , A m ) and r = (r it j) G are defined by 

1 

(2.19) A* = (A m , . . . , Ai); r* = curu, u 



so that 



(A*)j A m _j-|_i, 



fjij '"m— i+l,m— j+1- 



We have 

(2.20) r A * 
The relevant gamma function is defined by 



-1N-A 



r X ) A = rl A * 



/m 
r A e- tr < r »ir = 7r m(m_1)/4 T((A j - j + l)/2); 
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see, e.g., |FK1 p. 123]. The integral in (12.2ip converges absolutely if and 
only if Re Xj > j — 1 for all j = 1, . . . , m, and extends meromorphically 
to all A G C m . The following relation holds: 

(2.22) j r x e- tT ^d*r = T n (A) s; x % s G Q. 
h 

If A = (A, . . . , A) (€ C m ), then (cf. (123]) ) 

(2.23) r A ° = |r| A / 2 , T n (X ) = T m (A/2). 

2.3. The Radon transform on the space of matrices. The main 
references for this topic are |OR04j . [UR08aj . [UR08bj . [P70] . [ShT] . 
[Sh2j. Close results can be found in |GK3| IGra] . We fix positive in- 
tegers k, n, and m, < k < n, and let V n> k be the Stiefel manifold of 
orthonormal /c-frames in M n . For u G and t G 9Jtfc jm , the linear 
manifold 

(2.24) r = r(w, t) = {x G OT n , m : w'x = t} 

is called a matrix (n — k) -plane in 9Jt niTn . We denote by 1 the set of all 
such planes and consider the Radon transform 

(tt*/)(T) = / f(x), 

that sends a function / on £0T nim to a function 72.^/ on T. Precise 
meaning of this integral is the following: 

(2.25) (K k f){u,t)= j fU 
where g u G SO in) is a rotation satisfying 

(2.26) 5f u M = u, u = 

The next statement is a matrix generalization of the projection-slice 
theorem. It links together the Fourier transform (12 .ip and the Radon 
transform (I2.25P . In the case m — 1 , this theorem can be found in [Na| 
p. 11] (k = 1) and IS p. 283] (any 1 < k < n). 

Theorem 2.6. ( |OR05bl ) For f G L l (<m n , m ) and I < m <k, 

(2.27) = [F(K k f)iu, •)](&), it G & G OT fc , m 
where T stands for the Fourier transform on 9Jtfc jm . 





h 



G K 



n,k- 
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2.4. Riesz potentials and the Cayley-Laplace operator. The Riesz 
distribution h a on 9Jl n)m is defined by 

(2.28) (h a , f) = a.c. ^— / \x\™f(x)dx, f G S(Tt n , m ), 

. , 2 Qm vr nm / 2 r m (a/2) 

(2.29) 7n, m (a) = — r 77 _ W9 x — , a/n-m+l, n-m+2, . . .. 

For i?e« > in — 1, the distribution /i a is regular and agrees with the 
ordinary function h a (x) = \x\^ n /j n ,m( a )- The Riesz potential of a 
function / G 5(9tt„ im ) is defined by 

(2.30) (I a f)(x) = (h a , f x ), /„(•) = /(* - •)• 

For i?ea > m — 1, a/n - m + 1, n — m + 2, . . . , (I2.30p is represented 
in the classical form by the absolutely convergent integral 

(2.31) (rV)(x) = [ f(x - y)\y\ a m - n dy. 

This integral operator is well known in the rank-one case m = 1 |Ru96|, 
[MMllSa2llStT] . 

The Cayley-Laplace operator A on the space 9Jt„ jm is defined by 

(2.32) A = det(<9'«9). 

Here d is an n x m matrix whose entries are partial derivatives d/dxij. 
In the Fourier transform terms, the action of A represents a multipli- 
cation by the polynomial (— l)' m |?/|^. 

Theorem 2.7. [Ru06l Theorem 5.2] 

Let f G S(ffl njm ), aGC, a^n - m + 1, n — m + 2, Then 

(2.33) (h a ,f) = (2n)- nm (\y\- a ,(Ff)(y)), 

(2.34) (-l) mfc A fc fc Q+2fe = fc Q , /-2fe = (_i)^ A fc. fc = 0,l,2,... . 
Thus, one can formally write 

(2.35) / a = [(-l) m A]~ Q/2 . 

The next statement generalizes (12.331) to the case of composite power 
functions. 

Lemma 2.8. Let </> G 5(9Jl„ jm ). Then for all X G C m , 

(2-36) / - { y' y)X r JJWy)dy = c x [ (X ' X) ; ^) ^ 

y r n (A + no) y r n (-A*) 

n,m < $y^ n , m 

w/iere c A = 2" m +l A lvr nm / 2 , |A| = Ai + . . . + A m . 
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Remark 2.9. Both sides of (I2.33H and ( 12.36ft are understood in the 
sense of analytic continuation. Integrals in (I2.36P converge simultane- 
ously when A = (Ai, . . . , A m ) belongs to the set £ = £ fl £, where 

£ = {A : Re Xj > j — n — 1 for each j = 1, . . . , m}, 
£ = {A : Re Xj < j — m for each j — 1, . . . , m}. 

The diagonal Ai = . . . = X m = X does not belong to £. This explains 
essential difficulties when one tries to prove (12.331) directly as in [Ru06j . 
Formula ( I2.36f) was established by Khekalo |Khlaj . who extended the 
argument from [St If Chapter III, Sec. 3.4] to functions of matrix ar- 
gument. Khekalo's proof was reproduced in [OR05aj p. 61]. 



3. The Higher-Rank Funk Transform 



3.1. Definitions and duality. The classical Funk transform on the 
unit sphere S n ~ 1 C W 1 is defined by 



(3.1) (Ff)(u) 



f(v) d u v, 



ueS 



n-l. 



{?je.S n - 1 :u-?J=0} 



see, e.g., |GGGt IHellOj . We suggest the following generalization of 
(13.11) . in which u G V n ^ and v G V n ^ m are elements of respective Stiefel 
manifolds, 1 < k,m < n — 1. The higher-rank Funk transform sends a 
function / on V n m to a function F m k f on V n k by the formula 



(3.2) (F m , k f)(u)= / f(v)d u v, 

J {v€V n ,m - u'v=0} 

The corresponding dual transform 



u G V n . 



(3.3) (F m ,k<p){v) 



(f(u) d v u, 



veV r 



{u£V n , k :v'u=0} 



n,mi 



acts in the opposite direction. The condition u'v = means that sub- 
spaces uR k G G n ,k and v M. m G G n ,m are mutually orthogonal. Hence, 
necessarily, k + m < n. The case k = m, when both / and its Funk 
transform live on the same manifold, is of particular importance and 
coincides with (13. ip when k = m = 1. We denote F m = F m ^ m . 
To give the new transforms precise meaning, we set G = 0(n), 



(3.4) 
(3.5) 



K 



tEG:t 
peG : p 



7 

h 

5 

Im 



, 7 G 0(n - k) 
, 5 G 0(n — m 
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" " 




' h ' 











M = 


Ik 


, u = 





; = 




, ^0 = 






u ,u g V nik ] v ,v e V n>n , 

Then (13. 2p and (13.31) can be explicitly written as 



(3.7) (F mtk f)(u)= I f(g u 



to 




d*w= / f(9uTV )dr, 

'K 



(3.8) (F m , k (f)(v) 



V 1 9v 

Vn — mh 








dj= I ip(g v pu ) dp, 

'Ko 



where g u and g v are orthogonal transformations satisfying g u uo = u 
and g v vo = v, respectively. 

Remark 3.1. Since the measure d*u is right 0(m)-invariant, then, for 
all u G V n k, 



(3.9) 



(F m , k f)(u) = (F mik f)(u), f(v) = / f(vi) d 1 

'O(m) 



(similarly for (Fm,kV)( v ))- Hence, the set of all / G L 1 (\4 m ), for which 
f(v) = a.e., is a subset of ker F mtk ; cf. Theorem ll.il 

Lemma 3.2. Let 1 < k, m < n — 1; k + m < n. Then 

(3.10) / (F mjk f)(u) tp(u) d*u = /(f) (fm^lWiw 

provided that at least one of these integrals is finite when f and if are 
replaced by \ f\ and \ip\, respectively. 

Proof. We write the left-hand side as 

(F myk f)(gu )ip(gu )dg = / <p(gu ) dg / f(grv )dT. 
G Jg Jk 

Let k G 0(n) be such that u = xu and denote 

I m 

(3.11) C= 7 n _ m _ fc 

h 
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Keeping in mind that (v = v , ('u = u , and K is a stabilizer of v Q , 
we continue: 



l.h.s 



' <p(gxu )dg / dp f(gr(p'vo)dT 
G Jka Jk 



f(Xvo) dX dp (p(Xp('T xu ) dr 

G JKo Jk 

(note that ('t'xu = ('t'u = ('u = u ) 



f(Xv )d\ / (f(Xpu )dp 

JKo 

f(v) (F m ,k<p)(v)d*v, 



54.,., 



as desired. 



□ 



Corollary 3.3. If f £ L l {y ntm ), then the Funk transform {F m ^f){u) 
exists as an absolutely convergent integral for almost allu G V n ,k- More- 
over, 

(F m ,kf)(u) d*u = f(v)d*v. 
A similar statement holds for the dual transform F m ^- 



Example 3.4. Let (f(u) = \v' uu'vq 



l7/7//?>j( a - fe )/ 2 







, u e V n k. If 



l<m<k<n — m, Re a > m — 1, then 

(3.12) (F m ,k<p)(v) = c a \I m - v' vv'v \ {a ~ h)/2 , v e V n 

r m ((n-m)/2) r m (a/2) 



Tjk/2) T m ((a + n-m-k)/2Y 

Proof. The condition m < k has a simple explanation: if k < m then 
\v' q uu'vq\ = 0. Let us prove fl3TT2|) . By ([378]) . 

" 



(F m ,fcV?)(w) 

We write 

(3.13) g' v v = 



n — m,k 



A 
B 



\z e z' g \ {a - k)/2 d*d, z e = v' g v 



A G 9tt B G 

^ ^ ~'-' l ri— m,mi - tJ c ~'-' l m,m5 







so that = t4'#. Then we represent A in polar coordinates A = wS 1 ^ 2 , 
S = A' A eV m , we K-m, m - This gives 



(F m ,k<f)(v) 



\A'66'A\ {a - k)/2 dj = c a \S\ {a - k)/2 , 



n — m,k 
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where 

c a = ! \w'66'w\ {a - k)/2 d*0 

can be computed using formula (18.161) (with n replaced by n — m and 
A by a — k). Let x = ^ n ~' m £ V n ^ n - m . Then, by ( 13. 131) . A = x 'g' v Vo 
and 

S = A' A = v' g v KX > g' v v = v' g v (I n - v v' )g' v v = I m - v' vv'v . 
This gives the result. □ 
Example 13.41 and duality (I3.10p yield 

(F mtk f)(u)\v' uu'v \^- k ^ 2 d,u = c a [ f(v)\I m -v' vv'v \^- k ^ 2 d,v. 



n.k "Vn,m 

Owing to 0(n)-invariance, one can replace vq in this formula by an 
arbitrary w G V n . m . This gives the following statement. 

Lemma 3.5. Let w e V n , m , Re a > m — 1, 

(3.14) l<m<k<n — m. 

Then 

{F mM f){u)\w'uu'w\ {a - k)/2 d*u = c a I f(v)\I m ^w'vv'w\ {a ~ k)/2 d*v, 

provided that the integral on the right-hand side is absolutely conver- 
gent. 

3.2. Connection with Radon transforms on Grassmannians. 

There is an intimate connection between the higher-rank Funk trans- 
form and the well-known Radon transforms for a pair of Grassmann 
manifolds by inclusion. The latter were studi ed by s everal authors, 



who used different methods; see, e.g., [GTlRl IGGS701 IGrfl EES iKal 
IF67llZElllZE2| . 

Suppose that f(v) and <f(u) are right 0(m)-invariant and 0{k)- 
invariant functions on V ntTn and V n> k, respectively. We define the cor- 
responding functions on Grassmannians by setting 

(3.15) f(t)=*f( T ) = f(v), Z = {v}eG n , m , r = {v} ± eG n , n _ m , 

(3.16) 0{()=V>{n) = (p{u), ( = {u}eG n>k , r ] = {u} ± eG n:n _ k , 
and consider the Radon transforms 

(3.17) (R m ^ k f)( v ) = J /(o d v z, (R m , n . k <p)(0 = jHv) km 
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(3.18) (i2 fc , n _ m 0)(r)= J<p(()d T (, (k,n-mh(0= J}(r)d c r. 

CCr tD( 

Here cf,^, d^rj, d T (, d^r denote the relevant probability measures. 
Then (J22D and flU]) imply 

(3.19) (F m , k f)(u) = (R m ^J)(r ] ) = (R k , n - m f)(0: 

(3.20) (F m ,kf)(v) = ( J R fe , n _ m ^)(r) = (R m ,n-k £)(0- 
These equalities hold under the standard assumption 

dim G n>m < dim G„ >n _ fc or dimG n _ fc < dimG ni „_ m , 
according to which it is usually assumed l<m<k<n — m. 

4. Cosine and Sine Transforms. Composition formulas 
Lemma 13.51 suggests to introduce the following integral operators: 



(4.1) 


(C,J)H = 


/ f(v) 


\v'uu'v\ {a ~ k)/2 d*v, 


(4.2) 




/ <p(u) 


\v'uu'v\ (a - k)/2 d,u, 


(4.3) 


(S^ k f)(u) = 


[ m 


\I m -v'uu'v\ {a+k - n)/2 d^v, 


(4.4) 




/ 

Jv nik 


\I m - v'uu'v\ {a+k - n)/2 d*u 




ueV njk , 




1 < m, k < n — 1. 



We call C^ k f and S^ k f the cosine transform and the szne transform 

* * 

of /, respectively. Integrals C^k^P an d 5^ fcV 9 are called the duaZ cosine 
transform and the ditaZ sine transform. The terminology stems from 
the fact that, in the case k = m = 1, when u and v are unit vectors, 

I / / | / \2 2 It ''11/ \2 -2 

|t> mm v | = (u • v ) = cos cj, \l m — v uu v\ — 1 — [u ■ v ) = sin w, 

where u is the angle between u and v ; see also [2] IABI IGRl IOR06[ 
IQR05al !Zh2] , regarding higher-rank analogues of the cosine transform 
in the language of Grassmannians. 

Remark 4.1. When dealing with operators (14. ip and (14.21) . we restrict 
our consideration to the case m < k, because, if m > k, then \v'uu'v\ = 
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for all v G V n>m and all u G V n ,k- Similarly, for ( 14. 3 \ and (I4.4p . we 

assume m < n — k, because, if m > n — k, then 

\I m — v'uu'v I = \I m — v'Prr u yv\ = |i/Pr{ u }xt>| = \v'uu'v\ = 

(here u is an arbitrary (n — fc)-frame orthogonal to {u}). The case 
k = n, when v'uu'v = I m , is also not interesting. Clearly, 

(4.5) (S^J)(u) = (C^ n _J)(u)= j f{v)\v'uu'v\^- k ))l*d*v. 

The case k — m, when k and k coincide with their duals, are 
of particular importance. In this case we denote 



(4.6) {M a f)(u) = / f(v) \v'uu'v\ {a - m)/2 d*v, l<m<n-l, 

(4.7) (Q a f)(u) = [ f(v)\I m -v'uu'v\^ a+m ~ n)/2 d,v, 2m<n, 



where u G V n ,m- As we shall see below, Q a serves (after suitable 
normalization) as a substitute for the Riesz potential operator in the 
framework of the corresponding Radon theory. 

The following statement gives precise information about convergence 
of integrals (jOjl -dO). 

Theorem 4.2. Let f G L 1 {V n>m ) , (p E L 1 (V nt k) , 1 < m, k < n - 1. 

(i) Integrals converge absolutely almost everywhere if and 
only if Re a > m — 1 . 

(ii) If 1 < m < k < n — 1, then 

(4.8) / (C«J)(u)d*u = a f f{v)d*v, 

(4.9) / (P^ k (p)(v)d m v = ci / p(u)tU, 

r m (n/2) T m (a/2) 

Cl 



T m (A;/2)r m ((« + n- £;)/2)' 
(hi) If 1 < m < n — k, then 



(4.10) / (S« fc /)( u )d, u = c 2 / /(^)^, 



(4.11) / = c 2 / <p(u)d*u, 

Jv n>k 

T m (n/2) r m (a/2) 



c 2 



r m ((n-fc)/2)r m ((« + fc)/2)' 
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Proof. Equalities ( 14. 8ft and ( 14. 9ft hold by Fubini's theorem, owing to 
Lemma El and fl57T5]l . The proof of (T4TTUD and (T4TTT]) is similar. It 

suffices to note that 

\I m - v'uu'v\ {a+k - n)/2 d*u = I \v'uu'v\ {a+k - n)/2 cUv 

for any u G V n>n ^ so that (18.161) is applicable. The validity of (i) 
follows from the proof of ([4T8]l - (j4TTB . □ 

Functions (C^ lk f)(u) and (S^ lk f)(u) are right 0(fc)-invariant. Sim- 

ilarly, (C mkf)^) anc ^ (^mfcV 9 )^) are right 0(m)-invariant. Hence, 
transformations (14.ip - (14.4|) actually take functions on the Stiefel mani- 
folds to functions on the corresponding Grassmann manifolds. If / and 
ip are right-invariant in the suitable sense, our operators actually act 
from one Grassmannian to another. 

Below we derive a series of formulas connecting cosine, sine, and 
Funk transforms. Similar formulas and their applications in the case 
m — 1 can be found in |Ru02t IRu08| . 

Theorem 4.3. Let f G L 1 (V r n , m ), 1 < m < k < n — m. 
If Rea>m — 1, then 

(4-12) C m ,kFm,kf =Fm,kP m ,kf — °a Q a+n ~ k ~ m f \ 

T m ((n-m)/2)T m (a/2) 
° a T m (k/2)T m ((a + n-m-k)/2)' 
If Re a > k — 1 , then 

(4.13) h, k F m>k f =F m ,kC^J = ~c a M a+m ~ k F m f, 

~ T m (m/2) r m (a/2) 

° a T m (k/2)T m ((a + m-k)/2y 

Proof. The equality C m k^m,kf = c a Q a + n - k ~ m f i n (|4.12|) mimics Lemma 
* * 
The equality C m ,k F ™,kf =F m .kC mM f holds by duality: 



(Fm,kC^J,oj) = (f,C^ k F m , k u) = c a (f,Q 



a+n—k—m 



= c a (Q a+n - k ~ m f,u), u;GC°°(K,m). 

Note that by Theorem EES Q a+n - k - m f G L^K.m), because the condi- 
tions k < n — m and i?ea>m-l imply i?ea+n-fc-m>m-l. 
To obtain (14.131) . we first write (I4.12p with k = m. This gives 

(4.14) M a F m f = r m ((n-m)/2)r ro (a/2) m 
1 ^ J T m (m/2)T m ((a + n-2m)/2) V J 
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Then we replace a by a — k + m in ( 14.14ft to get Q a+n k m f on the 
right-hand side and compare the result with ( I4.12p . □ 

Theorem 4.4. Let f G L l (V n ^ m ), Re a > m — 1. If k < n — m, 

u G V nt k, then 

(4.15) (F m f i .M a f) (u) = c nAm {a) (<r*~"7)(«)> 



T m {{n-k)/2)T m {a/2) 



T m (m/2) T m ((a + n-k- m)/2) ' 
If 2m <n,ud V nt m, then 

(4. 16) (F m M a f) (it) = (M a F m f) (u) = c n>m (a) (Q°+»-*V) (it) , 

r m ((7i-m)/2)r m (q/2) 
Cn ' mlttj T m (m/2) r m ((a + n - 2m)/2) " 

Proof. By (13. 7p . denoting / u = / o g u , we obtain 



(F m>k M a f)(u) 



(M a f u ) 



n—k,m 



to 




/ d*u / f u {v) 



to 




d*v. 



Set v 



a 
b 



a = zs 1/2 , zeV n ^ k ,m, s = a'a = v'a a' v, a 
Changing the order of integration, we have 



a G M n _k,m, b G fflk,m and write a in polar coordinates 

G V n n —f~. 



(F m>k M a f)(u) = / f u (v)d*v 



In-k 





\u'a\ a - m d*u. 

-k,m 

The inner integral equals c|s|( Q ~ m " 2 , where 

r m ((n-fc)/2) r m (a/2) 



V'. 



T m (m/2) r m ((a + n-k- m)/2) ' 
see (18.16p . Hence, 

(Fm,kM a f)(u) = c! f u (v)\v'a a' v\^ m ^ 2 d,v 



c / f(v)\I m -v'uu'v\ (a - m)/2 d^ 



as desired. Equality (14.161) follows from (I4.12p and (I4.15p . 
The following useful factorizations hold. 



□ 
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Corollary 4.5. Let f G L 1 ^,™)- 

(i) If 2m < n, Re a > n — m — 1, then 

Q a f = d a M a+2m ~ n F m f = d a F m M a+2m ~ n f, 



(4.17) 



d n 



T m (m/2) F m {a/2) 



r m ((n - m)/2) r m ((a + 2m - n)/2) ' 
(ii) If m < k, Re a > k — 1, and u G V n ±, then for any u G {u} , 

(4.18) (C«J)(u) = d a (F m , n _ k M a+m ~ k f)(u), 

~ T m (m/2) r m (a/2) 

a r m (A;/2) T m ((a + m - k)/2) ' 

Proof, (i) can be obtained from (14.161) if we replace a by a + 2m — n. 
To prove (ii), we re- write (14.151) using (I4.5p . Then we replace k by n — k 
and u by u. This gives 



(C: fc fc -"7)(«) = ^ (^ m ,n- fc M a /)(«) 



u G {m}" 



T m (m/2) r m ((a + fc-m)/2) 



T m (A;/2)r m (a/2) 
Now we replace a + k — mby a, and we are done. 



□ 



Formula (I4.18P was obtained by the author several years ago and 
reported to S. Alesker, who gave another proof of it; cf. [2] Propo- 
sition 1.2]. A similar factorization in the language of Grassmannians 
is presented in \Zh2\ Lemma 3.4], however, without proof and without 
explicit constant. 

The following statement extends Theorem 14.31 to a = 0; cf. |Grit 
Theorem 2.4], where the close result was obtained in the language of 
Grassmannians in the spectral form. 

Theorem 4.6. Let 1 < k, m < n — 1; 2m < n — k. Then 

(4.19) F M = -cQ^f, a. ^~r m ((,-*)/2) 



r m (n/2) V m {{n-k-m)/2Y 



Proof. Let g v be an orthogonal transformation which sends v q 
to v G V n>m . We denote f v {w) = f(g v w). By (13. 2p and (13.31) . 







(F m ,kF m ,kf)(v) 



{F m ,kf){gvpu ) dp 



K 



dp 



K 



0{n-k) 



fv pC 



7 
h 



o?7, 
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where ( 6 O(n) is defined by ( 13.111) and satisfies ('uo = uq. Hence, 



(F m ,kF m ,kf)(v) 



d5 



0(n—m) 



fv 



5 

I'm. 



w 




d*w. 



Using the bi-Stiefel decomposition (12.121) (replace n by n — k, and k by 
m), the last expression can be written as 

1 



du. 



d5 / dv(r) / c?7 

®n-k,m J 0(n-m) Jo Jv m ,m 



< I fv 



6 

I m 



uilm-r) 1 ' 2 




c 

dv[r) I d^f I du 

Cn-k,m Jo JV m , m ^V n _ fc _ m , m 



X / fv 

' 0(n— m) 





a 



1/2 



d<5 



C "n—k—m.m I i / \ I j 

— / dv\r) / d*y 

&n—k,m Jo Jv m .m 



X / fv 

'v n 



n — m,m 



B{I m -rf^ 



d0. 



Here 



di/(r) = 2- m |rr /2 - d |/ m -r| (n - fc - m)/2 - d dr, 2m<n-fc. 



Now we change variables r — > I m — r and use (12.121) (with replaced 
by n — m) in the opposite direction. We obtain 



(F m ,kF m ,kf)(v) 



&n-k- 



@n—k,m J V n n 
&n—k—m,m @n,m 

®n—k,m Jv, 



f v (w)\w'(I n - v v' Q )w\ k/2 dw 
f(w)\I m - w'vv'w\~ k/2 d*w. 



Owing to (14.71) and (12. 8p . this is exactly what we need. 



□ 



Remark 4.7. The assumption 2m < n — k in Theorem 14. 61 is necessary 
for absolute convergence of the integral on the right-hand side in (I4.19P , 
whereas the left hand-side is finite a.e. under the weaker assumption 
m < n — k, which is sharp. By Lemma 15.131 below, the condition 
2m < n — k can be eliminated if we interpret (I4.19P in the iS'-sense. 
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5. Cosine Transforms via the Fourier Analysis 

In this section we proceed to develop the theory of the Funk, co- 
sine and sine transforms using the Fourier analysis in the ambient ma- 
trix space. The consideration essentially relies on the idea of analytic 
continuation, when the one-dimensional exponent a is replaced by a 
vector- valued complex parameter A G C m . 

5.1. The composite cosine transform. Let A = (Ai, . . . , A m ) G C m 
and let (-) A be the corresponding composite power function associated 
with the cone f2; see Section 12.21 We consider the composite cosine 
transform 



(5.1) (T^ m (f)(v) = I (p(u) (v'uu'v) x d*u, v G K,m, 

Vn.k 



which was studied in |OR06l IOR05aj when k = m. The dual cosine 
transform (14.21) is a particular case of (15. ip . corresponding to Ai = . . . = 
A m = oc-k. 

Lemma 5.1. Let l<m<k<n — 1, (p & L l {V n ^), 

(5.2) £ = {A G C m : Re \j > j — k — 1 for each j = 1, . . . , m}. 

The integral (T^ m ip)(v) converges absolutely for almost all v G V n , m 
if and only if A G £ and represents an analytic function of A in this 
domain. Moreover, 

5-3 / (T k (p){v)d,v = —————— — - / <p(u)d*u. 

Jv n , m r m {k/2)T Q (X + n ) J Vnk 

This statement follows immediately from Lemma 18.41 in Appendix. 

5.2. The complementary Radon transform. This is a new trans- 
formation that does not occur in jOR05al 1QR06] in the case k = m. It 
takes a function (p(u) on V n ,k t° a function (Ak !m (p)(v) on V n , m by the 
formula 



(5.4) (A km <f)(v) = <f[g v 



n — m.k — m 



a 

l<m 



d*a 



(F k - m ,m^([-,v}))(v), l<m<k<n-l, 



where g v is an orthogonal transformation that sends Vq 





I'm 



to 



v G V n>m . This can be regarded as a "partial Funk transform" (cf. 
(13.71) ). where ip is integrated over all orthonormal A;- frames u in M. n , 
the last m columns of which are replaced by v, and the first k — m 
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columns are orthogonal to v. If k = m, then A^ m reduces to the 
identity operator. 

Lemma 5.2. Let 1 <m <k <n — 1, ip £ L (V^&). Then 



(A k , m ip)(v) d*v = / <p(u)d*u. 



Proof. 



l.h.s 



dg I r - ! // 

O(n) JO(n-m) 





5 
I m 



O(n) 



dp = r./i.s. 




4 



d8 



□ 



5.3. The basic functional equation. The following lemma from 
[OR06j will be needed. We present it with proof in view of its fun- 
damental importance. We denote 

(5.5) A = {XeC m : Re\j>j-n-l for each j e {1, . . . ,m}, 

(5.6) A = {AeC m : Xj = j - n - I for some je{l,...,m} 

and some I G {1, 3,5,... }}. 

Lemma 5.3. Let f G L l {y n ^ m ), G S(Wt n ^ m ). The integrals 



(5-7) (r x f, 



r x f(v)<p(x)dx, (</,</>) = J < f{v)4>{x)dx 

*V in.rn *V *-tj 



are absolutely convergent if and only if A G A and extend as meromor- 
phic functions of A with the polar set A . The normalized integrals 



(r x f, 



{r X fA) 



(5 ' 8) r n (A + no)' r n (A + n )' 

n = (n, . . . , n), extend as entire functions of A. 

Proof. Consider the first integral. We set x = vt, v G V n>m , t G T m , 
and make use of Lemma 12721 By taking into account that x'x = t't and 
t{t't)' 1/2 G 0(m), owing to ( 12TTB|) . we obtain 



(5.9) 



„ m 

V, O) J /•(/,.;. • • • , /„,,„) tjf^dtjj , 
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where 

F(t ltl , . . . , t m , m ) = J dU J f{v)4>{vt) dv, dU = Y\ dt i,j- 

R m(m-l)/2 V n , m l<j 

Since F extends as an even Schwartz function in each argument, it can 
be written as . . . , t m>m ) = F (tl A , t 2 mm ), where F G S(R m ) 

(use, e.g., Lemma 5.4 from [TrJ P- 56]). Replacing $j by Sjj, we 
represent f 15 . 9 j) as a direct product of one-dimensional distributions 

m 

(5.10) (r x f, 0) = (n(^i)+ i+ ^ _1)/2 > *b(*i,i, • • • , V.)). 

i=i 

It follows that the the first integral in (15. 7p is absolutely convergent 
provided Re Xj > j — n — 1, i.e., A G A. The condition A G A is strict. 
To see this, we choose / = 1 and 0(x) = e _tr ^ ^ so that 



y (x'x) A e~ u ^ dx = 2- m a„, m r Q (A + n ) 



(5.11) (r A /,0) = 

Furthermore, since (sjj)+ ,+n 5 2 extends as a meromorphic distri- 
bution with the only poles Xj = j — n — l,j — n — 3, . . . , then, by the 
fundamental Hartogs theorem [ShaJ, the function A — > (r x f, (ft) extends 
meromorphically with the polar set A . By the same reason, a direct 
product of the normalized distributions 



( Aj+n _y_l)/ 2 



7r((A J + n-j + l)/2) 

is an entire function of A. 

Let us consider the second integral in (15.71) . Changing variable 
x = yu), where u is a matrix from ( I2.19p . we obtain r* = {x'x)* = 
(uy'yu)* = y'y. Hence (set y = ur, u G V ntTn , r G T m ), 

(r^fA) = j (y'y) X f(yu(uy'yu)- 1 ' 2 )Mwjdy 



/rn 

J'=l 



where, as above (note that rw (ujt'tuj) l ' 2 G O(m)) 



$(Tl,l, • • • , T m ,m) = j dn j f(u) (f)(uTu) du 

]Jm(m-l)/2 V n ,m 
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This gives 

m 

(r*f, 0) = {\{{s hJ ) { ^ n ' 3 ' l)/ \ $oKi, • • • , V)). 

and the result follows as in the previous case. □ 

We introduce the following normalized extensions of (Ak tm tp)(v ) and 
(T£ m <p)(v) from V n , m to the ambient matrix space 9Jt njm : 

(5.12) (A* (p) (x) = * (A k , m <p)(v), 

J- n{— A* J 

(5.13) (T A m ^)(x) = - (T A m y)(^) ; 

where x G 9# n ,m, # = t>r 1//2 , r = x'x 6 Q, d 6 V n ,m- 
The next theorem plays a key role in the whole paper. 

Theorem 5.4. Let (p be an integrable right 0(m) -invariant function 
on V n) m, G S{pi n ,m), l<m<k<n — 1. Then for every A G C m , 

r > nm+\\\ nm/2 p / /p\ 

(5-14) (T fc >^0)=c(^,0), c=± r m (fc/2) ' 

where both sides are understood in the sense of analytic continuation. 

Proof. It suffices to show the following. 

(a) Equality (15. 14ft holds for all A in a certain domain D C C m , where 
both sides of (I5.14p are analytic functions of A; 

(b) The right-hand side of (15. 14j) extends from D to all A G C m as 
an entire function. 

Let £> = £□£, where 

£ = {A G C m : Re Xj > j — k — 1 for each j = 1, . . . , to}, 
£ = {A G C m : Re Xj < j - m for each j — 1, . . . , to} 
(note that 2) is nonempty !). Absolute convergence of the integral 



(Al m¥ >,4>)= / (Al m <p)(x)<f>(x)dx 

for all A G £ and extendability of this expression by analyticity to all 
A G C m follow from Lemma [5.31 owing to Lemma [5.21 just change the 
notation in the second integral in (15 .7p and make use of the equality 
(A*)j = A m _ J+ i (the latter gives Re Xj < j —to). Absolute convergence 
and analyticity of the left-hand side of (I5.14p for all A G £ follow from 
Lemma [5731 too, thanks to Lemma I4~2l (note that £ C A, cf. (15. 5p ). 
Thus, it remains to prove (15.141) for A G 33. 
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Assuming A G £, by ( 12.181) we have 
(5.15) I=(f£ tm <p,F<t>) = [ v {u)J{u)d*u, 



J(u) 



rVA+ko) 



Vn,k 



xuux) ( J-"0) (x) dx, u E V n: k- 



M n , 



Let g u be an orthogonal transformation that sends Uq 





/a. 



to u e 



We set x = g u 
J(u) 



a 
b 



, a e Wln-fe,m, b e m,m- Then, by (12725]) . 



IVA 



^-y f (b'b) x db J (?</>)( g u I )da 
Mi Mi 



1 



Tn(A+k ) 



u 'b) x (n k F<f)){u,b) db. 



M k 



Owing to ( I2.36P and Remark 12.91 (we recall that AGS)!), the last 
expression can be written as 

r ,_ A) / (b'b); x *- k0 (J"- 1 ^^ -))(b)db 



M k , 



2|A|_-fem/2 

r a (-Aj 



(b'b)*° (HK k F<P)(u,-))(b)db. 



M k , 



By the Projection-Slice Theorem (see (I2.27P ). 

(5.16) (^(TZ k T(j))(u, •))(&) = {TT(j)){ub) = (2n) nm <f)(-ub). 

Hence, 

2im+| \\^m(n— k/2) 



J(u) = c x Jib'b) 
M km 



/I 1 ^-A„-ko■ 



(ub) db, c\ ■ 



Tof — A* 



This calculation enables us to transform ( I5.15P as follows: 
I = c\ [ (p(u)d*u I \b'b)Z x *-*°J(vti) db 



Mi 

2" m c A / r; A *- ko |r| (fc - m - 1)/2 ^(r)rfr, 



28 



B. RUBIN 



where 



dr\ J (p(u) 4>{urjr 1 / 2 ) d*u (set i] 



k,rn ) 



(?k,m J d'y J ipfaMwyrior 1 / 2 ) d*u 
0(k) v n<k 



= o~k,m J <f(u) (fiiurjor 1 / 2 ) d*u. 
Using notation from (13. 5p and (13.61) . we continue 



^(r) = cr fe , m / ip(gu )(j){gu r] r 1 l 2 )dg 

0(n) 

(note that u ?7o = Vq = 



e V n>m and K vq = v Q ) 



@k,m 

0{n) 



J dp J ^(gu )<j)(gp'v r 1 / 2 )dg 

K O(n) 



°k,m j 4>{\vQr l l 2 ) dX J (f(Xpuo) dp 

0(n) K 
®k.m 



(5.17) 



er r , 



(j){vr l l 2 ) dv 





' 5 ' 




' ' 




V \ 9v 


I m 






) 



dd 



0(n—m) 



cr,, 



^(vr 1 / 2 ) (A Km Lp)(v)dv. 



Hence (use (I2.17P and note that |r| = |r*|) 



2 m a r . 



c J (A k<m if)(x) (j)(x) dx, 



as desired. 



□ 



2!) 



The following corollaries hold in the case Ai = . . . = X m = a — k. 
We denote 



(5.18) (E x f)(x) = \r\ x/2 f(v), x = vr 1/2 G £Dl 



n.rn ■ 



Corollary 5.5. Let if be an integrable right O(k) -invariant function 
on V n ,k, 4> <S(9Rn,m), l<m<k<n — 1. Then for every a G C, 

,. I E a -k C ~r- i \ „ ( E k _ a _ n A km ip 

( 5 - 19 ) ,T<p \ = ci 



r m (a/2) J Vr m ((^-a)/2) 

2 m(n+ a -fc) ^nm/2 T m (n/2) 

° 1= f\^72) ' 

where both sides are understood in the sense of analytic continuation. 

In particular, if k = m, the following statement holds for the cosine 
transform 

(M a f)(u)= ! f{v)\v'uu'v\ (a - m)/2 d,v. 

Corollary 5.6. Let f be an integrable right 0(m) -invariant function 
on V n}jn , (j) G S(Wl n)m ), l<m<n — l. Then for every a G C, 

fr nn \ ( E a _ m M f \ I E m _ a _ n f 

( 5 - 20 ) fT~7 /on ' S<f> = C 2 ' 



T m («/2) ' v Vr m ((m-«)/2)' 

^ 2 m(n+a-m) r m (n/2) 

r m (m/2) 

where both sides are understood in the sense of analytic continuation. 

Remark 5.7. For better understanding of (I5.19P (and also (I5.2(jp ) we 
note that the domains, where the left-hand side and the right-hand side 
of this equality exist as absolutely convergent integrals, have no points 
in common, when m > 1. To implement analytic continuation we had 
to switch from a G C to A G C m . 

Corollary 5.8. Let if be an integrable right O(k) -invariant function 
on V n ^, l<m<k<n — 1. Then for any u G C°°(V njm ), the function 

/ 1 * 

vr m («/2) 

extends to all a G C as an entire function. 

Proof. Let -^(r) be a nonnegative C°° function with compact support 
away from the boundary of the cone Q. Choose J-(j) in (I5.19P so that 
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(J-(f))(x) = ip(r)u(v), x = vr 1//2 , v G V n>m . Then for Rea > m — 1, the 
left-hand side of (15.191) becomes (*(a)h^(a), 

h^a) = 2~ m [ \r\ {n+a - m ~ k - 1)/2 iKr)dr. 
Jn 

This gives 

5.21 a.c. C* (a) = a.c. — — , . 

ft^(a) I r m (a/2) / 

Owing to uniqueness of analytic continuation and Corollary I5.5[ the 
right-hand side of (I5.2ip is well defined and independent of ip. □ 

The following statement holds by duality. 

Corollary 5.9. Let f be an infinitely differentiable right 0{m) -invariant 
function on V n ^ m , l<m<k<n— 1. Then for any u> G C°°(V njJn ), 
the function 

« a ' = (rjb) c ^ w ) 

extends to all a G C as an entire function. 

5.3.1. Remarks and conjectures. 

1. We believe that Corollary 15.91 holds for every / G L 1 {y n>m ). This 
would follow from our next conjecture. 

2. It is natural to expect that for infinitely differentiable / and </?, 
the normalized functions 

(5-22) a -» 1 (C° */)(«), « -+ r r 7^ (C^OO 
r m (a/2) r m (a/2) 

extend as entire functions of a pointwise, that is, for every u and v, and, 
moreover, these extensions are C°° functions of u and v, respectively. 
We suppose that the proof of this fact can be given by using decompo- 
sition in Stiefel/Grassmann harmonics (see, e.g, [Gel IStr75l IStr861 ITT] 
for this theory). 

3. Analytic continuation of integrals, like C^ k f, was briefly dis- 
cussed in [GGRl p. 368], where it was suggested to replace such inte- 
grals by those over the matrix space SJl m ,n-m, so that dim9Jt min _ m = 
m(n — m) = dimG nim . However, the corresponding calculations were 
skipped in that paper. Our impression is that the actual procedure is 
more complicated; see Section 18.11 for further discussion. 
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5.4. A functional equation for the dual Funk transform. The 

next theorem specifies Corollary 15.51 for a = 0. According to this 

* 

theorem, the dual Funk transform F m ,k can be regarded as a member 
of the analytic family {C^k) ^ the latter is suitably normalized. 

Theorem 5.10. Let if be an integrable right 0{m) -invariant function 
on V n<m , <p G S(%R ntm ), l<m<k<n— 1. Then 

(5.23) (E_ k Fm.k^P, T<$) = c (E k „ n A k:m ip, 0), 

where A kjTn is the operator \5.4\ ), 

_ _ 2 m ("~ fc ) 7r nm / 2 r m ((n - k) /2) 
c ~ r m (fc/2) ' 

Proof. Passing to polar coordinates, we obtain 

* f * 

(E_ k Fm,k¥i F4>) = 2~ m °~n,m / (Fm,k¥)( V ) h{v) d*V, 



h(v)= / (^)(wV2) | r |(«-fc-"»- i)/ 2 ^. 



Hence, by duality (|3.10j) . 

(5.24) {E_ k Fm,W,J r <P) = 2- m / ip(u)(F m>k h)(u) d*u, 



where, by (I3.7p . 
{F m)k h)(u) = 



h g u 



V' 







2 m 







1/2 ] | r | (n-fc-m- 1)/2 ^ r 



0~n—ky 



- J {T<f>)(g u l^jdy 
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The last expression can be regarded as analytic continuation of the 
Riesz distribution ( I2.28|) . so that 

2 m 1 f 

{F m ,kh)(u) = a.c. -— / (TZ k J z (j))(u,z)\z\^ n dz. 

Now fl2T33|) yields 

(F m , k h)(u) = a.c / P[{R k F<P){u,-)]{y)\y\- m a dy. 

Hence, by (I5.16p . 

Qm(l— fc+n)^_m(n— k) r 

{F niyk h)(u) = / <p{uy)dy. 

0~n—k,m J 

Using (I5.24p and passing to polar coordinates, we obtain 

{E-k F m ,kV,F4>) = — / v( u )d*u j (p(uy)dy 

0~n—k,m J J 



(r) \nm-km 

a n,rr, / | r | ( k -m-l)/2 ^ ^ 



2 m (T , 

^ u n—k,m 

n 



where, by (I5.17p . 



Ok: 



H r ) = — — / 0(^ 1/2 ) (A kjm (p)(v) dv. 

&n,m J 

Setting x = vr 1 ^ 2 , we arrive at (I5.23p . as desired. □ 

Corollary 5.11. Let ip be an integrable right O(k) -invariant function 
on Ki.fc; <S( < ^n,m), l<m<k<n— 1. Then 



(5.25) (£_ fc F m ,kP, 0) = do a.c. 



E a -k C" k (f 



d, 



o 



a =6 I r m (a/2) 
r m {(n-k)/2)T m (k/2) 



r m (n/2) 

Proof. The statement follows immediately from (I5.23P and (I5.19p . □ 
In the case k = m we have the following 
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Corollary 5.12. Let f be an integrable right 0(m) -invariant function 

on Vn,m, £ <S(9Rn,m), 1 < W < 71 — 1. Then 

(5.26) (E^ m F m f, 7(f)) = c (£ m _„/, 0), 

2 m(n-m) _ m ) ^) 



Co 



r m (m/2) 



5.5. Analytic properties of the sine transform. Corollary 14. 5[ 
combined with analytic properties of the cosine transform, enables us 
to study analytic continuation of the sine transform 

(Q a f)(u) = [ f(v) \I m -v'uu'v\^ +m - n ^ 2 u G V n , m . 



Lemma 5.13. Let f be an integrable right 0(m) -invariant function 
on V n<m ; 2m < n, u G C°°(V^ im ). Then (Q a f,u) extends as a mero- 
morphic function of a with the polar set {m — 1, m — 2, . . . } by the 
formula 

(5.27) (Q a f, u) = d a (M a+2m - n F m f, u), 

T m (m/2) T m (a/2) 



T m ((n - m)/2) T m ((a + 2m - n)/2) ' 
The normalized function 

(5-28) r,(a)= 1 /0 A Q a f,oj) 

1 m {a/2) 

extends as an entire function of a. 

Proof. By fl4TT7D . for Rea> n-m-1 we have Q a f = d a M a+2m ~ n F m f. 
Hence, 

(QOlf c(M a+2m - n F m f,oo) = T m {m/2)T m {a/2) 

W hUJ} T m ((« + 2m-n)/2)' ' " T m ((n - m)/2) ' 
Applying Corollary 15. 81 with (p = F m f and k = m, we get the result. □ 

Remark 5.14. We conjecture that rj(a) is an entire function even if / 
is not right 0(m)-invariant, as in the case m = 1; cf. |Ru02t p. 474]. 

6. Normalized Cosine and Sine Transforms 

For the following it is convenient to suitably normalize cosine and 
sine transforms. The normalized transforms will be denoted by the 
corresponding calligraphic letters. Assuming l<m<k<n — 1, we 
set 

A. For u G V n h, veV„ m : 
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(6.1) CC, fc /)(«) = / /(*) \v'uu'v\^ 2 d*v, 

(6.2) (il k ip)(v) = 8 n!m;k (a) [ V {u)\v'uu ! v\^ 2 d^ 

(6.3) (yiJ)(u) = d n ^ k (a) [ f{v)\I m -v'uu'v\^ +k -^ 2 d^ 



(6.4) (S fi ° k <p){v) = d n , m , k {a) / v(u)\I m -v'uu'v\( a+h ~ n V 2 d* 



r m (m/2) r m ((fc-g)/2) 

^n, m ,t« = r ( /o\ — r 7 7^ — ' « + A; ^ 1,2,...; 
r m (n/2) T m (a/2) 

r m (fc/2) r m ((n - fc - g)/2) 

4,m,fc« = r p /0 . r p /0 . , a+k+m-n^ 1,2,...; 

r m (n/2) r m (a/2) 

B. For uG^m, «eK, m : 

(6.5) (JZ a f){u) = 5 n , m {a) [ f{v)\u'v\ a - m d*v, 



(6.6) (J2 a f)(u) = d n>m (a) / /(^IJ^W^I^-^ 2 ^, 2m<n; 



_ r m (m/2) r m ((m-a)/2) 
Vmlaj r m (n/2) r m (a/2) ' °^ ^> " " " < 

r m (n/2)r m (a/2) 

cf. Remark 14.11 By Theorem I4.2[ all these integrals are absolutely 
convergent if Re a > m — 1. Excluded values of a belong to the polar 
set of the corresponding gamma function in the numerator; cf. (12. 6p . 

Theorem 6.1. (i) Let ip be an integrable right O(k) -invariant function 
on V njk , u> G C°°(V^ im ), l<m<k<n — 1. Then 

«=o r m ((n-/c)/2) 

In particular, for any integrable right 0(m) -invariant function f on 
V nm , 2m < n, we have 

(6.8) a.c(^ a f,u)=fi m {F m f,u), ^ m =—^f— . 

a=o r m {{n-m)/2) 
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Proof. Equality fUD follows from ljQ5]) ; (ET25]) . and (JE2]); (EH]) is a 
consequence of f)6.7p . □ 

The following statement holds by duality. 

Corollary 6.2. Let f be an infinitely differentiable right 0(m) -invariant 
function on V n) m, l<m<k<n — 1. Then for any uj G C°°(V nim ), 

(6.9) a.c ■ (^ ik f,u)=fi k (F m>k f,u). 

Theorem 16.11 and Corollary 16.21 show that the Funk transform F m ^ k 
and its dual can be regarded (up to a constant multiple) as members 
of the corresponding analytic families of normalized cosine transforms. 
If m = 1, then (I6.7p - fl6.9l) agree with |Ru08l Lemma 3.1]. 

Lemma 6.3. Let f be an integrable right 0(m) -invariant function on 
V n ,m, l<m<k<n — m, 

Re a > m — 1, a ^ k — m + l,k — m + 2, .... 

Then 

a+ „_ fc _ m ^ _ _ T m ((n - m)/2) 



(6.10) V" k F m , k f = K k B a+n - k - m f, K k 



T m {k/2) 



Proof. Write (14. X2 j) in terms of normalized operators and you are done. 

□ 

The next theorem contains inversion formulas for the Funk trans- 
form F m and the cosine transform ^ a in terms of distributions. The 
corresponding results for m = 1 are due to Semyanistyi |Se63j . 

Theorem 6.4. Let f be an integrable right 0(m) -invariant function 
on V n ^ m , 2m < n. If Re a > m — 1, a ^ m, m + 1, m + 2, . . then 

(6.11) n a.c. (^^ a f,u) = (f,u), u;GC°°(K,m). 

T m ((n-m)/2) 



-2m— a— n 

Moreover (cf. ( 16. 8p ). 



(6.12) a.c. (Jt a F m f,u) = x rn (f,uj), x„ 

a=2m—n 

Proof. We write ( 15. 20ft in the form 

(6.13) ( E "-7f"^A=cJ Em-13-n.g 



r m (m/2) 



r m (/3/2) ' V \r m ((m-p)/2y 

2 m(n+l3-m) („ /2) 

C2= r m (m/2) ' 
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and set g = M a f. By Theorem 14.21 g is integrable and right 0(m)- 
invariant on V n m . Then we compute analytic continuation at the point 
(3 = 2m — a — n and obtain 



((2m-a-n)/2) ' / V r m((«+^- m )/ 2 ) ' 

2 m( TO -a) f nm/2 Y m (jl/2) 



C>2 



r m (m/2) 

By (15.201) the right-hand side of (16.141) can be written as 

c 2 r m (a/2) fE a ~ m M a f A \_ ( E m _ a _ n f __ v 



r m ((a+n-m)/2) V r m (a/2) 'V Vr m ((m-a)/2) 
where 

(27r) wt "r^(n/2)r m (a/2) 
3 r^(m/2)r m ((a + n-m)/2)' 

To finalize calculations, we set (J r ~ 1 0)(x) = (2n)~ mn (J r (j))(— x) and 
invoke normalizing factors according to ( 16.51) . This yields 

(6.15) (E m ^ n Jt 2m - a - n Jt a f, Tcj)) = (E m ^ n f, T<j>), 

where both sides are understood as analytic continuations from the 
respective domains. As in the proof of Corollary 15. 8[ we choose J-"</> in 
(I6.15P so that (J-"0)(x) = ip(r)u(v), x = vr 1 ^ 2 , v G V n>m . We recall that 
u) G C°°(V ntrn ) and i/j(r) is a nonnegative C°° function on the cone Q 
with compact support away from the boundary dQ. Then the left-hand 
side of ( 16.151) becomes 

a.c. (Ep_ m Jif l3 JZ cl f,ipu) = h^(a) a.c. {Jt^ 1 Jt a f , u), 

/3=2m—a—n f3=2m—a—n 

h^a) = 2~ m [ \r\- {a+1)/2 i^r) dr, 



Jn 

and the right-hand side equals h^{a) (f,u>). Owing to uniqueness of 
analytic continuation, this gives the result. 

Let us prove (16 .121) . We set g = F m f in (16.131) . Clearly, g is integrable 
and right 0(m)-invariant; cf. Corollary 13.31 Then we compute analytic 
continuation at the point = 2m — n and make use of Corollary 15.121 
This gives 

a a n c - i ~^Ta^~ ^ = d ° { E -mE m f, <P) = coo (E m -nf, F<f>), 

/3=2m-n \ L m {/j / 2) J 

. 2 m2 7c nm / 2 T m (n/2) . r m (n/2) 

°° r m (m/2)r m ((n-m)/2)' C °° r^(m/2)' 
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Hence, after normalization, 
a.c. (E /3 _ m ^^F m f,J r <f)) = >c m (E m _ n f,J r (f)), K m = Tm ^ n — ^V 2 ) 

/3=2m-n L m {m/2) 

As above, this implies ( 16.121) . □ 

The following important statement shows that the identity opera- 
tor is a member of the analytic family {J2 a } of the normalized sine 
transforms, corresponding to a = 0. 

Theorem 6.5. Let f be an integrable right 0(m) -invariant function 
on V ntrn ; 2m < n, u G C°°(y ntm ). Then (J2 a f,cu) extends as a mero- 
morphic function of a with the polar set {n — 2m + 1, n — 2m + 2, . . . }. 
Moreover, 

(6.16) a.c ■ J£ a f,tu) = (f,u). 

Proof. The statement follows from (15.271) and (I6.12p if we use normal- 
ization (16. 6p and (16. 5ft : 

' r m (n/2) r>=i™-» Vr m (/3/2)' 



o=0 



1 m\[n — m)l 2) fi=2m-n 



□ 



Equalities (I6.10P and (I6.16P imply the following inversion result for 
the Funk transform F m>k . 

Theorem 6.6. Let f be an integrable right 0(m) -invariant function 
on Vn,m, w G C°°(^n,m); l<m<k<n — m. Then 

(6-17) a.c (^ k F m , k f,u) = MM, ^= Tm r n 7ui /2) ■ 

a=m+k—n L m [K/Z) 

It would be natural to find an inversion formula for the cosine trans- 
form k > m, similar to (16. lip . Regretfully it is not available, 
however, the following lemma enables us to reduce inversion of k 
to the already known procedure for F m ; cf. (16.121) . 

Lemma 6.7. Let f be an integrable right 0(m) -invariant function on 
Vn,m, u e C°°(K, m ), 1 <m < k < n - m, 

Re a > k — 1, a k, k + 1, k + 2, ... . 

Then 

T m (m/2) 



(6.18) a.c. (yi k V" k f, u) = /i fc (F m f, u 
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Proof. By (Q51) and (1631) . 

^ a+ m-k Fmf = c p mk ci k f, c = 5 n , m (a + m - A;)/c a . 
Hence, by ( KTTTj) . 

(F m f,u) = a.c. c{Jt® F m , k C mk f,uj) 

p=m—n+k—a 

= a.c c6 n>m (P) ((F m>h MPyC« ;k f, u), 

p=m—n+k—a 

and fl4T5|) yields 

(F m f,uj) = a.c c8 n>m (/3) c nAm (P) {S^' k ~ m C^ h f,u). 

p=m—n+k—a 

Now we replace operators on the right-hand side by their normaliza- 
tions (16 .4p and (16. IB . A simple calculation completes the proof. □ 

The following "pointwise" conjecture looks natural. 

Conjecture 6.8. Letf(v) be an infinitely differentiable 0(m) -invariant 
function on V n ^ m . 

(a) For every v G V n ,m, (^ a f)( v ) extends as a meromorphic function 
of a with the polar set N = {1, 2, 3, . . .}. 

(b) For every complex a N, (^ a f)(v) is an infinitely differentiable 
0(m) -invariant function on V njTn . 

(c) If a (£ N U N, N = {2m - n - 1, 2m - n - 2, . . .} ; tfien 

(6.19) (^# 2m - a - n ^ a /)(v) = /(v) 

/or every «eF„ 

(d) T/ie transform 

(6.20) (F m f){u)= I f(v)d u v, ueV n , m , 

J {V &V n ,m. u'v=0} 

can be inverted by the formula 

(6.21) (^ 2m - n F m f)(v) = x m f(v), 

which holds for every v G V n ^ m . 

In the case m = 1 these statements are known and can be obtained 
using decomposition in spherical harmonics; see, e.g., |Ru98t IRu08] . 
We expect that Conjecture 16.81 can be proved using harmonic analysis 
developed in \Qe\ ITT] together with results from |Su] . 
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7. The method of Riesz potentials 

We already know that the Funk transform can be formally inverted 
as F" 1 = x^ 1 a.c. ^ a F m f . There are many ways to realize this 

a=2m—n 

analytic continuation. One of them amounts to Blaschke, Radon, Fu- 
glede, and Helgason |Rad| |Fug[ IHellO] and consists of two steps: (a) 



We apply a certain back-projection operator and reduce the problem to 
inversion of the corresponding potential; (b) We invert that potential 
operator. In the case of the Funk transform on the sphere, the potential 
operator is realized as the sine transform, explicit inversion of which is 
pretty sophisticated; see, e.g., |HellOt IRu02] . In the higher-rank case 
the situation is much more complicated; cf. \Gn\ Theorem 3.4], \K&\ 
Theorem 10.4]. Semyanistyi [Se63] suggested to express the spherical 
sine transform through the spatial Riesz potential. In this section we 
generalize his idea for the higher-rank case. 

There is a substantial difference between the cases, when n — k — m 
is an even number and an odd number. This phenomenon, which is 
well-known in the case m — 1, is related to the so-called local and 
non-local inversion formulas. 

7.1. Local inversion formulas for the Funk transform. It is in- 
structive to start with the particular case k = m, when evenness of 
n — k — m is equivalent to the evenness of n. 

Theorem 7.1. Let f be a right 0(m) -invariant function in L 1 (V^ l)m ), 
2m < n. If n is even and 

(—1 ) m ( n /2-m) 2 m ( 2m - n ) y 2 (771/2) 
° m= Tl((n-m)/2) m ' 

then the Funk transform (p = F m f can be inverted by the formula 
(7. 1) E m _ n f = c m A"/ 2 ~ m E_ m F m ip (in the S'-sense) . 

If the right-hand side of l[7.1\ ) is locally integrable in the neighborhood 
of the Stiefel manifold V n ^ m , then \1. 1\ ) holds pointwise a.e. on V ntTn . 

Proof. By (1qT2T)1) for e S(DJl n , m ) we have 

(-l) m \A £ E^ m F mV , <P)=d {E m . n¥ >, |a&) = c {E m „ n+2f F m f, 0). 

We can choose 2£ = n — 2m to get c (E_ m F m f, (ft). By (I5.26P this 
coincides with Cq (2n)~ nm (E m _ n f, <fi), and the result follows. □ 



The next statement is more general. 
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Theorem 7.2. Let f be a right 0(m) -invariant function in L 1 (V riirn ), 
l<m<k<n — m. If n — k — m is an even number and 

(-l)"* 4 -"*r m (m/2)r m (fc/2) 
Cm ' fc " r m ((n-m)/2)r m ((n-fc)/2)' " l " 

then the Funk transform if = F m ^f can be inverted by the formula 
(7.2) E m _ n f = c mtk A E_ k F m ,kV (in the S"-sense). 



If the right-hand side of {1.2$ is locally integrable in the neighborhood 
°fVn,m, then {7TM) holds pointwise a.e. on V n>m . 



Proof. By (15251) and (Q51) . 



-ir\A*E_ k F m ^A) = d a.c n I ^±g|^ >( -i)^ 

I r m (a/2) 



, / E a _ k M a+m k F m f ,\mt\l' 

Vr m ((a + m- A;)/2)' v ; 

, r m ((n - fc)/2) r m (m/2) 
oti — 



r m (n/2) 

Owing to (I5.20p . for 2£ = n — k — m, this expression can be written as 
rf 2 o-c. (E k „ n _ a F m f, = d 2 (E_ m F m f, (ft), 



a=0 



d 2 = 2 m ^ vr" m / 2 r TO ((n - k)/2)/T m (k/2). 
Finally, by QQgp , 

(-l)^(A^_ fe F m ,^,0) = c d 2 (E m . n fJ) = (E m . n fJ). 

This gives the result. □ 

7.2. Sine transforms and non-local inversion formulas. Theo- 
rems 17.11 and 17.21 show that when n — k — m is odd, we have to deal 
with fractional powers of the operator A. The latter are realized by 
the Riesz potential I a ; see Section [2^1 For the following we need some 
more preparation. 
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7.2.1. The Semyanistyi-Lizorkin-Samko spaces <&y. From the Fourier 
transform formula (12.331) it is evident that the Schwartz class S is 
not well-adapted for Riesz potentials, because S is not invariant under 
multiplication by \y\^- To get around this difficulty, we choose another 
space of test functions. Let ^ = \I/(9Jt„ ;m ) be the collection of all 
functions ip(y) G <S(50t n)TO ) vanishing on the manifold 

(7.3) V = {y G M n;m : rank(y) <m} = {ye aH n , m : \y'y\=0} 

with all derivatives. The manifold V is a cone in M. nm with vertex 
0. Let $ = $(££Jtn jTO ) be the Fourier image of Since the Fourier 
transform is an automorphism of S, then $ is a closed linear subspace 
of S, which is isomorphic to ^f. 

The spaces $ and \I> were introduced by V.I. Semyanistyi [Se] in the 
case m = l. They have proved to be very useful in integral geometry 
and real analysis. Further generalizations and applications are mainly 
due to Lizorkin and Samko; see (LTJ, IRu96l ISal| ISKMJ on this subject. 

The following characterization of the space $ is a consequence of a 
more general result by Samko |Salj . 

Theorem 7.3. The Schwartz function <p{x) on ffl n , m belongs to the 
space $ if and only if it is orthogonal to all polynomials p(x) on any 
hyperplane r in lR nm having the form r = {x : tr(a'x) = c}, a G V : 



d/j,(x) being the induced Lebesgue measure on r. 

When m = 1, the space $ consists of Schwartz functions which are 
orthogonal to all polynomials on IR n . 

We denote by $' the space of all semilinear continuous functionals 
on $. Two ^'-distributions that coincide in the $'-sense, differ from 
each other by an arbitrary iS'-distribution with the Fourier transform 
supported by V. Since for any complex a, multiplication by is 
an automorphism of then, according to the general theory |GSh2j . 
I a is an automorphism of $, and we have 



(7.4) 





F[I«4>}(y) = \y\- a T[4>}(y), 



ni a f]{y) = \y\- m a nf]{y) 



(f) G $. 



for any ^'-distribution /. For k even, the Riesz potential I k f can be 
inverted by repeated application of the Cayley-Laplace operator A in 
the sense of ^'-distributions. 
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7.2.2. Inversion of the sine transform. 

Lemma 7.4. Let f be an integrable right 0(m) -invariant function on 
V n ,m, % m < n. Then for any a G C, 

(7.7) a.c. E a+m _ n £ a f = 2 am a.c. I a E m . n f 
in the $' '-sense. 

Proof. Let ip — (j) G ^f. We first suppose that 

(7.8) Rea>n — m — 1; a^n— m, n— m+1, n— m+2, . . . . 
Then, owing to (|BTB) . (Q7|) . and (15301) . 

(£ Q+m _ n jr/,</>) = d ntm (a)d a (E a+m _ n M a+2m - n F m f,^) 

n(a+m)m nm/2 p / /o^ 

(7.9) = ^— -f ^^(^ m F m /,|x|->). 

r m ((n - m)/2) 

Since |z|- a ^(x) = (27r)- nm |x|- Q 0(-x) = (27r)- nm (/ Q 0) A (-a;), then fl5^6|) 
yields 

(E a+m ^ a f, 0) = 2 am (£ m _ n /, J» = 2 am (I a E m _ n f, 0). 

An expression in ( I7.9P is an entire function of a. Hence, the result 
follows by analytic continuation. □ 

Lemma [7.41 enables us to reconstruct / from Q a f in the ^'-sense. 

Corollary 7.5. Let g = Q a f, where f is an integrable right 0(m)- 
invariant function on V njm , 2m < n, 

Rea>m — 1; a^n—2m+l,n—2m+2,.... 

We set a = 21— 7, where i is a positive integer. The following inversion 
formula holds in the $' '-sense: 

(7.10) E m . n f = 2~ am (-lf m A e PE a+m . n Q a f, 
where A is the Cay ley- Laplace operator \2. Sty) . 

Remark 7.6. An analogue of (17. 7p for m — 1 is contained in Lemma 
3.1 from |Ru02] and invokes spherical convolutions with hypergeometric 
kernel. It is an interesting open problem to extend that Lemma to the 
higher-rank case. 
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7.2.3. Non-local inversion of cosine and Funk transforms. 

Theorem 7.7. Let f be an integrable right 0(m) -invariant function 
on V nm , 1 < m < k < n — m, and let £ be a positive integer. 

(i) Ifg = ^J, 

Rea>m — 1, a + m — k ^ 1, 2, . . . , 

and a + n — k — m = 2£ — 7, then the following inversion formula holds 
in the $' '-sense: 

(7.11) £ m -n/ = c A e r'E a _k F m ,k^,J, 

r m ((n-m)/2) 

(ii) If <p = F m k f and n — k — rn — 2£ — 1, then, similarly, 

(7.12) E m _ n f = C* A £ I 1 E_ k Frn,kFm,kf, 

_ 2 m ( fc+m - n )(-l) m£ r m (fc/2)r m (m/2) 
°*~ r m ((n-m)/2)r m ((n-fc)/2) 
Proof, (i) By f )4.12p . after normalization, we obtain 

a+n-k-m r ~ c a^n,m,k( a ) 



r m,k& m hJ — °a ~* J 



d n ,m(a: + n - fc - m) 

Then we apply Corollary 17.51 with a replaced by a + n - k — m. This 
gives the result. 

(ii) By ([5~19]) and ( 15T23]) . for any /i G 5(S0t n>m ) we have 

(7.13) a.c I ° k , Cr J l 'f > , h I = Ai F m , fe </?, /i), 

a=0 I r ro (a/2) / 

r m (n/2) 

^1 



r ro (A;/2) r ro ((n - fe)/2) 
Owing to (I4.12p . the left-hand side of ( 17. 13ft can be written as 

«=o r m (a/2) 

Now we replace h by the Riesz potential J 1_2 ^0 = (— 1) A where 
2£ — 1 = n — k — m, and apply (17.71) . This gives 

Ax (-l) m£ (A e I 1 E_ k F m , k <P, 0) = A 2 a.c. (I a E m . n f, 0) = A 2 (£ m _ n /, 0), 

ct=0 

2 m(n-m-AO rm ( 7l / 2 ) ^(fo _ m )/ 2 ) 

r^(fc/2)r m (m/2) 
and the result follows. □ 
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8. Appendix 



8.1. On the paper by Gelfand, Graev, and Rosu. Sections 2 
and 3 of |GGR] deal with analytic continuation of a certain auxiliary 
operator Rg; see, e.g., formula (3.5) in |GGR] . Ignoring the normalizing 
factor, and changing notation p = m, A = a + n — m, one can write it 
as a cosine transform 



(M a f)(u) 



f(v) \uv 



/„,| a— m 



where 2m < n, Re a > m — 1, and / is a right 0(m)-invariant func- 
tion. It was stated on p. 368, that elementary computation (which is 
skipped) leads to the formula of i? A in coordinates; see formula (3.6) 
in the same paper. Below we perform computation and arrive at a dif- 
ferent expression, which does not fall (at least, directly) into the scope 
of references [P67[ lRaij|. as stated in [GGRl Proposition 1, p. 368] B 

Let Mn-m^m be the subset of 9Jt n _ mjm , which consists of matrices of 
rank m. Every matrix y G ifl n ~m,m is uniquely represented in polar 
coordinates as y = us 1 

(8.2) v 

Since 



1//2 , u G V n - mm , sefi. We set 



u{{I m + s)-\s)^ 



(Im + S) 



-1/2 









. V2 . 



v'v 



((I m + s)' l sf/ 2 uj'u{{I m + sy's) 1 / 2 + (I m + s)- 1 
= (I m + s)-^ + (I m + s)- 1 = I m , 

then v G Ki,m- Thus (18.21) defines a map \i : 9Jt n _ mjm — >■ V n ^ m . Con- 
versely, given v G V ntTn , the corresponding matrix y = cos 1 / 2 G M r , 
can be reconstructed from obvious relations 



v'v v' Q v, 



(8.3) (I m + s)- 1 

u{{I m + s)- l s) l ' 2 = v [ 



v 



v 







l n—m 





G V n ^ m 
G V n 



Lemma 8.1. Let 2m < n. If f is an integrable right 0(m) -invariant 
function on V n ^ m , f(y) = (f o n){y), then 



U) 



f(v) dv 



Or, 



Vn 



Kv) 



dy. 



n — m,m 



2 The reasoning from P67 is reproduced in Lemma 
situation. 



+ y'y\ n / 2 

for the more general 
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If F is an integrable function on 9JT n _ mim; F(v) = (Fo|i 1 ){v), then 

1 f F(v) 



5.5) 



F(y) dy 

m ° 



+n — m,m 



m,m JV n ,m \ V V \ n 



dv. 



Proof. We denote by / the left-hand side of (E3D- By fl232|) with 

k = n — m, 



dv{r 
o v r 



dw 



f 



wr 



1/2 



u 



(Im-r) 1 / 2 



du. 



n — m,rn 



du(r) = 2- m \r\ {n - 2m - 1)/2 \I m - r\~ 1/2 dr. 
Changing variable r = I m — (I m + s) -1 , we get 

u;((/ m + S )- 1 S ) 1 /2 



dv(s) 



dw 



f 



Vm 



u(I m + s) 



-1/2 



du, 



du{s) = 2~ m \s\ {n - 2m - 1)/2 \I m + s\~ n/2 ds. 

Since / is right 0(m)-invariant, then integration over V m ^ m = 0(m) 
can be suppressed and Lemma 12.11 yields (18 .4p . The second equality is 
a consequence of the first one, owing to (18. 3p . □ 

Now we return back to (18. ip . In the statement below t>o, f, F, and 
F have the same meaning as in Lemma \S7L\ u, v G V n)Tn \ x,y G S371 
u = We set 

\I m + x'x\( m ~ a V 2 



n—m,m i 



K a (x,y) 



KJu,v) 





v' u 


m—a 


\v' v 


n+a—m 



\l m + y'y\ {n+a - m)/2 '' 

Lemma 8.2. The following relations hold provided that integrals in 
either side are absolutely convergent: 



(8.6) f(v)\u'v\ Q - m d*v-- 



f(y)\I m +x'y\ a - m K a (x,y)dy, 



n — m.m 



17) 



F(y)\I m +x'y\ a - m dy 



a„ 



f{v) \u'v\ a - m K a (u,v) d*v. 



971 



n — m.m 



Vn,: 



Proof. As above, we set x = Or 1 / 2 , y = us 1 / 2 , where 9, to G V„-m,m and 
r, s G Q. Then we define v G V n ^ m by (18. 2 j) and, similarly, 

^((/ m + r)^ 1 r) 1 /2 
(J^ + r)- 1 / 2 





Ml 




. U2 . 
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Keeping in mind that 

r(I m + r)- 1 = (J m + r)-V, ((J m + r^r) 1 / 2 = (J m + r^V 1 / 2 , 
we easily have 

\u'v\ = \u[vi + U2V2 1 

Hence, by (IQl . 



|/m + H- 1/2 |/ m + r^W' 2 ! |/ m + s\- 1 ' 2 . 



\u'v\ 





1 


m + x'y 




1 -^m ~\~ % % 


1/2 


Im + y'y 


1/2 



|im + x'y\ 





u'v 




\v' u 




v' v\ 



It remains to plug these expressions in the corresponding integrals and 
apply Lemma [8. II □ 



Remark 8.3. An important factor K a (x,y) in (18. 6p . that depends 
on a and affects the behavior at infinity, is skipped in [GGR| formula 
(3.6)]. Moreover, Lemma lBTTl reveals that, in general, f(y) is not rapidly 
decreasing. It follows that the reasoning from [P67t IRaij . related to 
analytic continuation of the distribution |x| A /T P ((A +p)/2) cannot be 
directly applied to ( 18. 6ft . To understand the essence of the matter, 
the reader is encouraged to perform analytic continuation in detail for 
m — 1. Note also that, the integral operator on the right-hand side of 
(18.71) is not Q(?T-)-invaria nt, unlik e the left-hand side of (18. 6p . 



In the preceding work [GGS70] , the result was deduced from [GGS67 
using transition to the Radon transform on the space of matrices. This 
transition requires careful inspection becaus e it does not lead to rapidly 



decreasing functions, which are assumed in [GGS67 
8.2. A useful integral. 

(Ai, . . ■ , A m ) g cr 



Lemma 8.4. Let X 

1 < m < k < n. Then 



19) 



(v'uu'v) x d*v 



(V uuv) 



u G V n>k , v G V n>m , 
ko) 

V m {k/2) r n (A + n )- 



T m (m/2)T n (X 



This integral converges absolutely if and only if Re Xj > j — k — 1 for 
each j = 1, 2, . . . , m. 



Proof. The first equality follows from Lemma 12.41 Both integrals are, 
in fact, constants. Thus, it suffices to evaluate 



(8.10) 



(v'u u' v) x d*v, w 



h, 




gK, 



k- 
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Consider an auxiliary integral 



(8.11) 



.4 



(x'u u' x) x e~ tv{x ' x) dx. 



We compute it in two different ways. Let first 

d G ^Ul krn , b G ^Xfi n —} i ^ rn . 



x 



a 
b 



Then u' x = a, x'x = a' a + b'b, and we have 



(8.12) 



A — AiA 2 , A x = J {a'a) x e- tr{a ' a) da, A 2 = J 



db. 



Passing to polar coordinates, owing to (12.211) and (I2.23p . we obtain 

(8.13) A x = 2- m a k>m J r A+k ° d*r = 2~ m a^ m Tq(X + k ), 

h 

provided Re Xj > j — k — 1, Vj = 1,2, ... ,m. The last condition is 
sharp. It gives the "only if part of the lemma. For A 2 we have 



m(n—k) 



An = \ / e~ s ds 



71 



m{n—k)/2 



Thus 



5.14) A 



^fiitA + kp 
Y m {k/2) 



ReXj > j - k — 1. 



On the other hand, by setting x — vt, v G Ki,m> £ € ^m, owing to 
Lemma [2.21 we obtain 



- tr( *' 4) d/x(t) / (t'v'u u' vt) x dv, 



rf/i(t) = t n -J dt jtj dU, dU = Y\ dtij. 

j=X i<j 

By (12.181) . one can write 

(8.15) A = BI, 
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where I is our integral ( 18.10P and 

B = a n , m J {t't^e-^dnit) 



1 1 n n 

j =1 -oo 

m \ j_ ■ I i 



= 2- m a n , m r Q (A + n ), ite A, > j - n - 1. 
Combining this with f !8.14p and f)8.15p . we obtain 
_ A X A 2 r m (m/2)r n (A + k ) 



B r m (£;/2)r Q (A + n ) 



□ 



In the case Ai = . . . = X m = X we have the following. 
Corollary 8.5. Let 1 < m < k < n, Re\> m — k — 1. Then 

, / , ,A/ 2 , /" | / / ,A/2 , r m (w/2)r m ((A+fc)/2) 

1 T m (A;/2)r m ((A+n)/2) 



(8.16) 
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